Lexicalized Hidden Markov Models for Part-of-Speech Tagging

Since most previous works for HMM-based tagging consider only part-of-speech information in contexts, their models cannot utilize lexical information which is crucial for resolving some morphological ambiguity. In this paper we introduce uniformly lexicalized HMMs for partof-speech tagging in both English and Korean. The lexicalized models use a simpli ed back-o smoothing technique to overcome data sparseness. In experiments, lexicalized models achieve higher accuracy than non-lexicalized models and the back-o smoothing method mitigates data sparseness better than simple smoothing methods.